Scaffolding rollout workspace multi worker#1011
Closed
narutolhy wants to merge 18 commits into
Closed
Conversation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add is_ipv6_address() to detect IPv6 address strings - Add format_addr() to bracket IPv6 in host:port strings (RFC 2732) - Extend gethostip() with IPv6 fallback for IPv6-only environments - Update is_port_free() to check both AF_INET and AF_INET6 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use format_addr() everywhere host:port strings are constructed so that IPv6 addresses are properly bracketed in URLs. Also: - remote_inf_engine: accept per-worker port_range to avoid concurrent find_free_ports() TOCTOU races when multiple workers start in parallel - rollout_controller: partition port space by worker rank so each worker picks from a non-overlapping range, eliminating bind conflicts Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Bracket IPv6 addresses in tcp://[addr]:port init_method so PyTorch dist.init_process_group can parse the URL correctly - Pass group=self.cpu_group (Gloo) to allocate_balanced_mbs_synced inside split_padded_tensor_dict_into_mb_list; without this it falls back to NCCL WORLD which hangs on nodes where IB has no routing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When multiple arun_episode coroutines run concurrently they previously shared mutable state on PipelineTrajectoryMaker (task_data, prompt_str, input_tokens), causing later episodes to overwrite earlier ones before generation completed. - ScaffoldingLlm.generate_async: clone prototype_controller synchronously before any async handoff so each episode gets its own isolated clone - ScaffoldingWorkflow.arun_episode: pass per-episode data as kwargs to generate_async instead of mutating shared trajectory_maker fields - PipelineTrajectoryMaker.process: pop task_data/prompt_str/input_tokens from kwargs so each cloned controller sees its own episode data Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GSM8KScaffoldingWorkflow.build_scaffolding_llm was constructing NativeGenerationController() with no arguments, leaving sampling_params empty. convert_task_params skips None fields, so SGLang never received max_tokens and defaulted to ~16 tokens, producing truncated completions and near-zero rewards (~0.3%). Fix: delegate to super().build_scaffolding_llm(engine) which correctly builds sampling_params from gconfig (max_tokens, temperature, stop). Also clean up debug instrumentation added during investigation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- cli_args: raise setup_timeout 300→1200s (SGLang model loading on large clusters takes >5 min); add sdpa/eager to attn_impl choices; use sys.executable instead of hardcoded python3 - ray.py: raise startup_timeout 30→600s for multi-node Ray worker join - gsm8k dataset: update prompt to request step-by-step reasoning - chat_scaffolding.yaml: update to 8-GPU xccl config - Add gsm8k_rlvr_scaffolding_2nodes.yaml for 2-node A100 Ray setup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly expands AReaL's capabilities by integrating a flexible Scaffolding framework, allowing for more sophisticated and composable reinforcement learning workflows. It enhances the system's ability to manage multi-worker inference and training, particularly for large language models, through improved networking and dynamic resource allocation. The changes lay the groundwork for advanced RL techniques like verifiable rewards and multi-turn conversational agents, making the platform more versatile and robust for complex AI research. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a significant new feature: the scaffolding framework for composing inference and training workflows. The implementation is robust, featuring a well-designed asynchronous architecture to prevent deadlocks, and includes comprehensive improvements for multi-worker and IPv6 environments. The code is well-structured, extensively tested with both unit and integration tests, and includes clear examples. I have reviewed the changes and found no issues of medium or higher severity; the quality of the implementation is excellent.
Note: Security Review did not run due to the size of the PR.
This was referenced May 14, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Related Issue
Fixes #(issue)
Type of Change
work as expected)
Checklist
jb build docs/gemini review)Breaking Change Details (if applicable):
Additional Context
Need help? Check the Contributing Guide or ask in
GitHub Discussions!